MADiff: Offline Multi-agent Learning with Diffusion Models

Diffusion model (DM), as a powerful generative model, recently achieved huge success in various scenarios including offline reinforcement learning, where the policy learns to conduct planning by generating trajectory in the online evaluation. However, despite the effectiveness shown for single-agent learning, it remains unclear how DMs can operate in multi-agent problems, where agents can hardly complete teamwork without good coordination by independently modeling each agent's trajectories. In this paper, we propose MADiff, a novel generative multi-agent learning framework to tackle this problem. MADiff is realized with an attention-based diffusion model to model the complex coordination among behaviors of multiple diffusion agents. To the best of our knowledge, MADiff is the first diffusion-based multi-agent offline RL framework, which behaves as both a decentralized policy and a centralized controller, which includes opponent modeling and can be used for multi-agent trajectory prediction. MADiff takes advantage of the powerful generative ability of diffusion while well-suited in modeling complex multi-agent interactions. Our experiments show the superior performance of MADiff compared to baseline algorithms in a range of multi-agent learning tasks.Comment: 17 pages, 7 figures, 4 table

RITA: Boost Autonomous Driving Simulators with Realistic Interactive Traffic Flow

High-quality traffic flow generation is the core module in building simulators for autonomous driving. However, the majority of available simulators are incapable of replicating traffic patterns that accurately reflect the various features of real-world data while also simulating human-like reactive responses to the tested autopilot driving strategies. Taking one step forward to addressing such a problem, we propose Realistic Interactive TrAffic flow (RITA) as an integrated component of existing driving simulators to provide high-quality traffic flow for the evaluation and optimization of the tested driving strategies. RITA is developed with consideration of three key features, i.e., fidelity, diversity, and controllability, and consists of two core modules called RITABackend and RITAKit. RITABackend is built to support vehicle-wise control and provide traffic generation models from real-world datasets, while RITAKit is developed with easy-to-use interfaces for controllable traffic generation via RITABackend. We demonstrate RITA's capacity to create diversified and high-fidelity traffic simulations in several highly interactive highway scenarios. The experimental findings demonstrate that our produced RITA traffic flows exhibit all three key features, hence enhancing the completeness of driving strategy evaluation. Moreover, we showcase the possibility for further improvement of baseline strategies through online fine-tuning with RITA traffic flows.Comment: 8 pages, 5 figures, 3 table

Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization

Recent progress in state-only imitation learning extends the scope of applicability of imitation learning to real-world settings by relieving the need for observing expert actions. However, existing solutions only learn to extract a state-to-action mapping policy from the data, without considering how the expert plans to the target. This hinders the ability to leverage demonstrations and limits the flexibility of the policy. In this paper, we introduce Decoupled Policy Optimization (DePO), which explicitly decouples the policy as a high-level state planner and an inverse dynamics model. With embedded decoupled policy gradient and generative adversarial training, DePO enables knowledge transfer to different action spaces or state transition dynamics, and can generalize the planner to out-of-demonstration state regions. Our in-depth experimental analysis shows the effectiveness of DePO on learning a generalized target state planner while achieving the best imitation performance. We demonstrate the appealing usage of DePO for transferring across different tasks by pre-training, and the potential for co-training agents with various skills.Comment: 22 pages, 3 tabels, 15 figure

Effects of Compound Feed Attractants on Growth Performance, Feed Utilization, Intestinal Histology, Protein Synthesis, and Immune Response of White Shrimp (<i>Litopenaeus Vannamei</i>)

To investigate the effects of compound attractants on the growth performance, feed utilization, intestinal morphology, protein synthesis, and immune response of Litopenaeus vannamei, the following seven diets were formulated: a positive control (P), a negative control (N), and five diets with compound attractants which were labeled as A, B, C, D, and E, each with four of five tested attractants (yeast extract, squid visceral powder, fish soluble, and squid paste, shrimp paste), respectively. Shrimp (0.71 ± 0.00 g) were distributed to seven groups of four replicates and fed for 7 weeks. Results showed that the final body weight, feed intake, specific growth rate, and weight gain of shrimp in the B and D groups were the greatest. Hemolymph total antioxidant capacity of shrimp in the B, D, and E groups reached the highest level. In the hepatopancreas, the activity of total nitric oxide synthase, malondialdehyde content, the expression levels of sod, myd88, eif4e2, and raptor in shrimp fed the B diet were the highest, and the highest levels of dorsal and relish were observed in the C group. In the intestine, intestinal muscle thickness and expression levels of toll and eif2α in the C group were the highest, while the highest expression level of sod and relish occurred in the B group. In summary, the B and E diets promoted the feed intake, growth performance and the antioxidant enzyme activity of L. vannamei. The C diet enhanced the protein synthesis of shrimp. Regression analysis indicated that the WG and FI of shrimp were increased as the dietary inclusion levels of squid paste and shrimp paste increased, while they were decreased as the dietary inclusion levels of yeast extract and fish soluble increased

Effects of Dietary Fish Meal Replaced by Fish Steak Meal on Growth Performance, Antioxidant Capacity, Intestinal Health and Microflora, Inflammatory Response, and Protein Metabolism of Large Yellow Croaker Larimichthys crocea

Although fish steak meal (FSM) is a potentially available protein source, its efficiency as a fish meal (FM) substitute remains unclear to date. To this end, this study was carried out to determine the effects of dietary FM replaced by FSM on growth performance, antioxidant capacity, intestinal health and microflora, inflammatory response, and protein metabolism of large yellow croaker. Five isolipidic and isonitrogenous diets were formulated by substituting FM with FSM at levels of 0% (FSM0, control diet), 25% (FSM25), 50% (FSM50), 75% (FSM75), and 100% (FSM100), and were fed to juvenile large yellow croaker for 8 weeks. Compared with the control diet, the replacement of 25% dietary FM with FSM did not markedly alter the weight gain (WG) and specific growth rate (SGR). When the FM substitution level was over 25%, WG and SGR markedly reduced. The intestinal structure observation found that the FSM75 and FSM100 diets markedly decreased villus height, villus width, and muscle thickness of the anterior intestine. The FSM75 and FSM100 diets significantly decreased enzyme activities of amylase (AMS), lipase (LPS), trypsin, catalase (CAT), and total superoxide dismutase (T-SOD) and the total antioxidant capacity (T-AOC), and increased the malondialdehyde (MDA) content in the liver of large yellow croaker. The mRNA expression levels of intestinal barrier and inflammatory response-related genes suggested that the FSM50, FSM75, and FSM100 diets significantly decreased the mRNA abundances of intestinal barrier-related genes and anti-inflammatory response-related genes, and increased the mRNA abundances of proinflammatory gene il-6 in the anterior intestine. The compositions of intestinal microflora displayed that the FSM50, FSM75, and FSM100 diets decreased relative abundances of Firmicutes phylum and increased relative abundances of Proteobacteria phylum. In addition, the results of protein expression levels showed that the phosphorylation level of mammalian target of rapamycin (mTOR) and 4E-binding protein 1 (4E-BP1) in FSM75 and FSM100 groups were markedly reduced. In conclusion, FSM can replace up to 25% dietary FM without compromising the growth performance, intestinal health, and protein metabolism of the large yellow croaker

Partial Substitution of Fish Meal with Soy Protein Concentrate on Growth, Liver Health, Intestinal Morphology, and Microbiota in Juvenile Large Yellow Croaker (Larimichthys crocea)

The present study investigated the growth performance, feed utilization, intestinal morphology, and microbiota communities of juvenile large yellow croaker (Larimichthys crocea) fed diets containing different proportions of soy protein concentrate (SPC) (0, 15%, 30%, and 45%, namely FM, SPC15, SPC30, and SPC45) as a substitute for fish meal (FM) for 8 weeks. The weight gain (WG) and specific growth rate (SGR) in fish fed SPC45 were significantly lower than those fed FM and SPC15 but not differ with these fed SPC30. The feed efficiency (FE) and protein efficiency ratio (PER) decreased sharply when the dietary SPC inclusion level was higher than 15%. The activity of alanine aminotransferase (ALT) and expression of alt and aspartate aminotransferase (ast) were significantly higher in fish fed SPC45 than those fed FM. The activity and mRNA expression of acid phosphatase were opposite. The villi height (VH) in distal intestine (DI) showed a significant quadratic response to increasing dietary SPC inclusion levels and was highest in SPC15. The VH in proximal intestine, middle intestine decreased significantly with increasing dietary SPC levels. The 16S rRNA sequences in intestine revealed that fish fed SPC15 had higher bacterial diversity and abundance of Phylum Firmicutes such as order Lactobacillales and order Rhizobiaceae than those fed other diets. Genus vibrio, family Vibrionaceae and order Vibrionales within phylum Proteobacteria were enriched in fish fed FM and SPC30 diets. Tyzzerella and Shewanella that belongs to phylum Firmicutes and Proteobacteria, respectively, were enriched in fish fed SPC45 diet. Our results indicated that SPC replacing more than 30% FM could lead to lower quality diet, retard growth performance, ill health, disordered intestine structure, and microbiota communities. Tyzzerella could be the bacteria indicator of intestinal in large yellow croaker fed low quality diet due to high SPC content. Based on the quadratic regression analysis of WG, the best growth performance could be observed when the replacement of FM with SPC was 9.75%